Skip to content

feat: implement endpoint to support fetching workspace pod log - #1267

Open
YoyinZyc wants to merge 4 commits into
kubeflow:notebooks-v2from
YoyinZyc:log_batch
Open

feat: implement endpoint to support fetching workspace pod log#1267
YoyinZyc wants to merge 4 commits into
kubeflow:notebooks-v2from
YoyinZyc:log_batch

Conversation

@YoyinZyc

@YoyinZyc YoyinZyc commented Jul 21, 2026

Copy link
Copy Markdown

Implement the batch log API proposed in #886 (comment)
closes: #887
related: #886

Behavior

A new one-shot endpoint returning a workspace pods container(including the initContainer like istio) logs as a plain text.

Query params: container(defaults to first/primary container); tail(default to 1000); previous (default to false); sinceTime(default to none)

API

GET /api/v1/workspaces/{namespace}/{name}/podtemplate/logs/batch
      ?container=<name>    # optional, defaults to the first (primary) container
      &tail=<int>          # optional, default 1000
      &previous=<bool>     # optional, default false (previous terminated instance)
      &sinceTime=<string> #optional, default to none

Response

2026-07-23T18:37:19.502427795Z 2026-07-23T18:37:19.502162Z	info	xdsproxy	connected to delta upstream XDS server: istiod.istio-system.svc:15012	id=96
2026-07-23T19:07:20.316168674Z 2026-07-23T19:07:20.315975Z	info	xdsproxy	connected to delta upstream XDS server: istiod.istio-system.svc:15012	id=97
2026-07-23T19:36:29.611434886Z 2026-07-23T19:36:29.611255Z	info	xdsproxy	connected to delta upstream XDS server: istiod.istio-system.svc:15012	id=98
2026-07-23T20:04:05.870345556Z 2026-07-23T20:04:05.870108Z	info	xdsproxy	connected to delta upstream XDS server: istiod.istio-system.svc:15012	id=99
2026-07-23T20:32:49.598960074Z 2026-07-23T20:32:49.598795Z	info	xdsproxy	connected to delta upstream XDS server: istiod.istio-system.svc:15012	id=100
2026-07-23T21:01:53.168136917Z 2026-07-23T21:01:53.167896Z	info	xdsproxy	connected to delta upstream XDS server: istiod.istio-sy

Unit tests

internal/repositories/logs/repo_test.go
workspace_logs_handler_test.go

Tested Manually

BASE_URL=https://localhost:8443/workspaces/api/v1/workspaces/default/jupyterlab-workspace/podtemplate/logs/batch

Positive

Path HTTP Code Result
empty 200 default the main container with tail set to 1000 and previous=false
tail=15 200 15 line of log
container=main 200 return log for main container
container=istio-proxy 200 return log for istio proxy
sinceTime=2026-07-23T18:30:00Z 200 return main log since that time
container=istio-proxy&sinceTime=2026-07-23T18:30:00Z 200 return istio proxy log since that time

Negative

Path HTTP Code Error Msg
previous=true(never restarted) 409 no logs found for the previous container instance
container= 400 container not found in pod
tail= / tail=0 / previous=maybe / sinceTime= 422 field validation error
non-existent workspace 404 workspace not found
non-running container(in waiting state) 409 container has not started yet (note: this one is hard to test manually, mostly rely on unit test)

@github-project-automation github-project-automation Bot moved this to Needs Triage in Kubeflow Notebooks Jul 21, 2026
@google-oss-prow google-oss-prow Bot added the area/backend area - related to backend components label Jul 21, 2026
@google-oss-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign andyatmiami for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow google-oss-prow Bot added area/v2 area - version - kubeflow notebooks v2 size/XXL labels Jul 21, 2026
@YoyinZyc
YoyinZyc force-pushed the log_batch branch 2 times, most recently from f184804 to fcb68f7 Compare July 21, 2026 19:50
@christian-heusel

Copy link
Copy Markdown
Member

@YoyinZyc FYI I have added a compression middleware to the backend in parallel which should help quite a bit for the log endpoint 🤗 #1269

@YoyinZyc
YoyinZyc force-pushed the log_batch branch 2 times, most recently from 0496f5d to f30cafe Compare July 25, 2026 01:04
@christian-heusel

Copy link
Copy Markdown
Member

/ok-to-test

YoyinZyc added 4 commits July 30, 2026 17:14
… workspace pod log

Signed-off-by: Yuchen Zhou <yczhou@google.com>
… level unit tests

Signed-off-by: Yuchen Zhou <yczhou@google.com>
Signed-off-by: Yuchen Zhou <yczhou@google.com>
Signed-off-by: Yuchen Zhou <yczhou@google.com>

@andyatmiami andyatmiami left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@YoyinZyc - Sorry I've been distracted by life - and took me awhile to be able to dig into this PR.

Overall very excited about the implementation here - a lot of "nitpicks" around various things that is completely natural this being your first pR..

But the core implementation is really well done

Let me know if you disagree with any of my comments and want to talk about them - always happy to discuss!

Comment on lines +188 to +199
// HTTP: 404 with a caller-provided message.
func (a *App) notFoundResponseWithMessage(w http.ResponseWriter, r *http.Request, err error) {
httpError := &HTTPError{
StatusCode: http.StatusNotFound,
ErrorResponse: ErrorResponse{
Code: strconv.Itoa(http.StatusNotFound),
Message: err.Error(),
},
}
a.errorResponse(w, r, httpError)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Open to discussion - but I'm not sure this function is really warranted.. all existing endpoints use the existing a.notFoundResponse(w, r) and I think that works well enough as is..

Would prefer to keep with convention here - but let me know if I am overlooking something and/or why you think we should add this...

if err != nil {
switch {
case errors.Is(err, repository.ErrWorkspaceNotFound):
a.notFoundResponseWithMessage(w, r, err)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related to: https://github.com/kubeflow/notebooks/pull/1267/changes#r3695881990

I think we should just stick with a.notFoundResponse(w, r) to "keep it simple"

var (
ErrWorkspaceNotFound = fmt.Errorf("workspace not found")
ErrPodNotRunning = fmt.Errorf("workspace pod is not running")
ErrContainerNotFound = fmt.Errorf("container not found in pod")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
ErrContainerNotFound = fmt.Errorf("container not found in pod")
ErrContainerNotFound = fmt.Errorf("container not found in workspace pod")

logsContainerQueryParam = "container"
logsTailLinesQueryParam = "tailLines"
logsPreviousQueryParam = "previous"
logSinceTimeQueryParam = "sinceTime"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
logSinceTimeQueryParam = "sinceTime"
logsSinceTimeQueryParam = "sinceTime"

Comment on lines +185 to +198
It("should return 409 when the workspace pod is not running", func() {
By("creating the HTTP request")
req, ps := buildLogsRequest(namespaceName, workspaceName, "")

By("executing GetWorkspaceLogsHandler")
rr := httptest.NewRecorder()
a.GetWorkspaceLogsHandler(rr, req, ps)
rs := rr.Result()
defer rs.Body.Close()

By("verifying status is 409 Conflict")
Expect(rs.StatusCode).To(Equal(http.StatusConflict))
})
})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 409 test ("workspace pod is not running") only checks the status code:

Expect(rs.StatusCode).To(Equal(http.StatusConflict))

But the 404 test in this same file goes further — it parses the ErrorEnvelope and asserts the error message matches the sentinel. The 409 test should do the same for consistency within the file as well as generally being much more reliable/robust/accurate.

if opts.Previous && apierrors.IsBadRequest(err) && strings.Contains(err.Error(), "previous terminated container") {
return nil, ErrPreviousLogsNotFound
}
return nil, fmt.Errorf("failed to open log stream for pod %s, container %s: %w", podName, containerName, err)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems like we can at least define the "template string" as a const at the top of this file - so its easier to see/locate all our error strings together..

Comment on lines +141 to +145
// None requested: default to the primary (first regular) container.
if len(podStatus.Containers) == 0 {
return "", "", ErrContainerNotRunning
}
containerName = podStatus.Containers[0].Name

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 I wonder if we capitalize on the fact we hardcode main as the container name for our primary container - and in this branch "force" that as the default...

i guess defensively we'd still want to ensure there IS a main container to be robust against the unknown

// if the target container is still in the Waiting state (i.e. it has not started yet
// and therefore has no logs available for the current instance).
func (r *LogsRepository) ensureContainerStarted(ctx context.Context, namespace, podName, containerName string) error {
pod, err := r.clientset.CoreV1().Pods(namespace).Get(ctx, podName, metav1.GetOptions{})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is probably implied from the function comment:

  • the live Pod status

... but might be nice to also explicitly call out here we are opting to use the clientset (vs. client) as we are (presumably) trying to make a real-time decision on state and want to avoid a potentially stale cached version

// The Workspace status references a pod that no longer exists.
return ErrPodNotRunning
}
return fmt.Errorf("failed to get pod %s: %w", podName, err)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems like we can at least define the "template string" as a const at the top of this file - so its easier to see/locate all our error strings together..

Comment on lines +177 to +188
for _, group := range [][]corev1.ContainerStatus{pod.Status.ContainerStatuses, pod.Status.InitContainerStatuses} {
for _, cs := range group {
if cs.Name != containerName {
continue
}
// A container that is still Waiting has never started and has no logs yet.
if cs.State.Waiting != nil {
return ErrContainerNotRunning
}
return nil
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While admittedly not a realistic "performance concern" - I personally like the "more boring" two-loop style iteration present here:

its a simple linear scan with no extra allocations...

but at minimum - I think we should be consistent in iteration logic implementation

@christian-heusel christian-heusel left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work @YoyinZyc this is already quite awesome 🔥

I looked through the code a bit and left a review below! Feel free to apply / challenge / ignore it as needed 🤗

Comment on lines +27 to +29
// The number of lines to retrieve from the end of the logs.
// By default, the value is 1000.
TailLines int64

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not currently seem to be respected:

$ curl -sk -H "Kubeflow-Userid: admin" https://localhost:8443/workspaces/api/v1/workspaces/default/jupyterlab-workspace/podtemplate/logs/batch\?tail=10 | wc -l 
42

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah nevermind, this was just due to me using the parameter naming:

$ curl -sk -H "Kubeflow-Userid: admin" https://localhost:8443/workspaces/api/v1/workspaces/default/jupyterlab-workspace/podtemplate/logs/batch\?tailLines=10 | wc -l 
10

Maybe we should guard against wrongly spelt parameter? 🤔 Anyways, that's most likely out of scope for this PR 😅

Also see https://github.com/kubeflow/notebooks/pull/1267/changes#r3708558872

Comment on lines +38 to +41
logsContainerQueryParam = "container"
logsTailLinesQueryParam = "tailLines"
logsPreviousQueryParam = "previous"
logSinceTimeQueryParam = "sinceTime"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think those should potentially live in api/constants/query_params.go:

const (
NamespaceQueryParam = "namespace"
NamespaceFilterQueryParam = "namespaceFilter"
)

Comment on lines +126 to +163
func parseLogOptions(r *http.Request) (*models.LogOptions, field.ErrorList) {
var valErrs field.ErrorList
query := r.URL.Query()

opts := &models.LogOptions{
Container: query.Get(logsContainerQueryParam),
}

if raw := query.Get(logsTailLinesQueryParam); raw != "" {
tail, err := strconv.ParseInt(raw, 10, 64)
if err != nil || tail <= 0 {
valErrs = append(valErrs, field.Invalid(field.NewPath(logsTailLinesQueryParam), raw, "must be a positive integer"))
} else {
opts.TailLines = tail
}
}

if raw := query.Get(logsPreviousQueryParam); raw != "" {
previous, err := strconv.ParseBool(raw)
if err != nil {
valErrs = append(valErrs, field.Invalid(field.NewPath(logsPreviousQueryParam), raw, "must be a boolean"))
} else {
opts.Previous = previous
}
}

if raw := query.Get(logSinceTimeQueryParam); raw != "" {
t, err := time.Parse(time.RFC3339, raw)
if err != nil {
valErrs = append(valErrs, field.Invalid(field.NewPath(logSinceTimeQueryParam), raw, "must be a valid RFC3339 timestamp"))
} else {
sinceTime := metav1.NewTime(t)
opts.SinceTime = &sinceTime
}
}

return opts, valErrs
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think to more closely follow the codebase conventions we could potentially introduce 3 new helper functions and add them to internal/helper/validation.go:

func ValidateFieldIsPositiveInt64(path *field.Path, value string) (int64, field.ErrorList)
func ValidateFieldIsBool(path *field.Path, value string) (bool, field.ErrorList)
func ValidateFieldIsRFC3339Time(path *field.Path, value string) (metav1.Time, field.ErrorList)

Also we could validate the container name:

if raw := query.Get(logsContainerQueryParam); raw != "" { 
  valErrs = append(valErrs, helper.ValidateFieldIsDNS1123Label(field.NewPath(logsContainerQueryParam), raw)...)
  opts.Container = raw
}

Comment on lines +102 to +103
case errors.Is(err, repository.ErrPodNotRunning):
a.conflictResponse(w, r, err, nil)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is a conflictResponse really the right thing to return here? 🤔
I'm asking because this is also what is returned in case a workspace is paused:

$ curl -k -H "Kubeflow-Userid: admin" https://localhost:8443/workspaces/api/v1/workspaces/default/jupyterlab-workspace/podtemplate/logs/batch
{"error":{"code":"409","message":"workspace pod is not running","cause":{}}}

// When previous=true but the container has never restarted, the Kubernetes
// API returns a 400 with a "previous terminated container ... not found"
// message. Surface this as a semantic error instead of a generic 500.
if opts.Previous && apierrors.IsBadRequest(err) && strings.Contains(err.Error(), "previous terminated container") {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked a bit into alternatives for it and since we already fetch the pod we can also do something like this (🤖 -generated; just want to showcase the idea):

diff --git a/workspaces/backend/internal/repositories/logs/repo.go b/workspaces/backend/internal/repositories/logs/repo.go
index f8a7917b..b37afd20 100644
--- a/workspaces/backend/internal/repositories/logs/repo.go
+++ b/workspaces/backend/internal/repositories/logs/repo.go
@@ -20,7 +20,6 @@ import (
 	"context"
 	"fmt"
 	"io"
-	"strings"
 
 	kubefloworgv1beta1 "github.com/kubeflow/notebooks/workspaces/controller/api/v1beta1"
 	corev1 "k8s.io/api/core/v1"
@@ -87,12 +86,6 @@ func (r *LogsRepository) OpenLogStream(ctx context.Context, namespace, workspace
 
 	stream, err := req.Stream(ctx)
 	if err != nil {
-		// When previous=true but the container has never restarted, the Kubernetes
-		// API returns a 400 with a "previous terminated container ... not found"
-		// message. Surface this as a semantic error instead of a generic 500.
-		if opts.Previous && apierrors.IsBadRequest(err) && strings.Contains(err.Error(), "previous terminated container") {
-			return nil, ErrPreviousLogsNotFound
-		}
 		return nil, fmt.Errorf("failed to open log stream for pod %s, container %s: %w", podName, containerName, err)
 	}
 	return stream, nil
@@ -145,47 +138,52 @@ func (r *LogsRepository) resolvePodAndContainer(ctx context.Context, namespace,
 		containerName = podStatus.Containers[0].Name
 	}
 
-	// When requesting current (not previous) logs, ensure the target container has
-	// actually started by inspecting the live Pod status. A container still in the
-	// Waiting state (e.g. PodInitializing, ContainerCreating, ImagePullBackOff) has
-	// no current log stream yet, and the Kubernetes API would return an opaque error;
-	// surface it as a semantic 409 instead. Previous logs are exempt, since a
-	// terminated instance can have logs even while the current instance is Waiting.
-	if !opts.Previous {
-		if err := r.ensureContainerStarted(ctx, namespace, podName, containerName); err != nil {
-			return "", "", err
+	// Inspect the live Pod status to determine whether the requested log stream can
+	// actually exist. Deciding this up front from the typed status lets us return a
+	// semantic error instead of interpreting the opaque error the Kubernetes API
+	// would otherwise return.
+	cs, err := r.findContainerStatus(ctx, namespace, podName, containerName)
+	if err != nil {
+		return "", "", err
+	}
+	if opts.Previous {
+		// A previous instance exists only if the container has terminated at least
+		// once. This holds even while the current instance is Waiting (e.g. a
+		// container in CrashLoopBackOff still has logs from its last run).
+		if cs.LastTerminationState.Terminated == nil {
+			return "", "", ErrPreviousLogsNotFound
 		}
+	} else if cs.State.Waiting != nil {
+		// A container still in the Waiting state (e.g. PodInitializing,
+		// ContainerCreating, ImagePullBackOff) has never started, so it has no log
+		// stream for the current instance yet.
+		return "", "", ErrContainerNotRunning
 	}
 
 	return podName, containerName, nil
 }
 
-// ensureContainerStarted checks the live Pod status and returns ErrContainerNotRunning
-// if the target container is still in the Waiting state (i.e. it has not started yet
-// and therefore has no logs available for the current instance).
-func (r *LogsRepository) ensureContainerStarted(ctx context.Context, namespace, podName, containerName string) error {
+// findContainerStatus returns the live status of the named container, searching both
+// the regular and init container statuses of the pod.
+func (r *LogsRepository) findContainerStatus(ctx context.Context, namespace, podName, containerName string) (*corev1.ContainerStatus, error) {
 	pod, err := r.clientset.CoreV1().Pods(namespace).Get(ctx, podName, metav1.GetOptions{})
 	if err != nil {
 		if apierrors.IsNotFound(err) {
 			// The Workspace status references a pod that no longer exists.
-			return ErrPodNotRunning
+			return nil, ErrPodNotRunning
 		}
-		return fmt.Errorf("failed to get pod %s: %w", podName, err)
+		return nil, fmt.Errorf("failed to get pod %s: %w", podName, err)
 	}
 
-	// Search both regular and init container statuses for the target container.
 	for _, group := range [][]corev1.ContainerStatus{pod.Status.ContainerStatuses, pod.Status.InitContainerStatuses} {
-		for _, cs := range group {
-			if cs.Name != containerName {
-				continue
-			}
-			// A container that is still Waiting has never started and has no logs yet.
-			if cs.State.Waiting != nil {
-				return ErrContainerNotRunning
+		for i := range group {
+			if group[i].Name == containerName {
+				return &group[i], nil
 			}
-			return nil
 		}
 	}
 
-	return ErrContainerNotRunning
+	// The container is declared in the pod spec but has no status yet, so it has not
+	// started and has no logs.
+	return nil, ErrContainerNotRunning
 }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/backend area - related to backend components area/v2 area - version - kubeflow notebooks v2 ok-to-test size/XXL

Projects

Status: Needs Triage

Development

Successfully merging this pull request may close these issues.

3 participants